Overview

Introduction to Statistical Inference

• “Statistical inference” describes a decision making process wherein statistics and probability theory are used to draw conclusions about some unknown aspect of a population

  • Is a core concept for experimental research
  • Has broad applicability across many domains
  • Can sometimes be difficult for students to understand

• Decision are made on the basis of a random sample of observations that were drawn from the population

• Statistical inference can be sub-divided into two main areas: Estimation and Hypothesis testing

Estimation

• In estimation, a random sample is used to describe some unknown aspect of the population from which the sample was taken

  • Measures of central tendency - mean
    - median
    - mode
    
  • Measures of spread/variability - variance
    - standard deviation
  • Parameter values - shape parameter
    - scale parameter
    - rate parameter

• These descriptions come in one of two types - depending on the goal of the analysis

  • Point estimates
  • Interval estimates

Hypothesis Testing

• In hypothesis testing, a random sample is used to decide which of two complementary statements about a population are true

  • A hypothesis is a tentative explanation for something
  • Hypothesis testing begins by drawing a random sample and computing on or more characteristics (statistics)
  • Hypothesis testing is an inferential process, based on probability, and is used to draw conclusions about the population parameters

Motivating Example

Building Widgets

• Suppose that we manage a factory that produces “widgets”

• Each widget we produce comes with a lot of measureable properties

  • Physical properties: Strength, hardness, fracture toughness
  • Electrical properties: Conductivity

• For each widget, the values associated with each of these properties is based on the materials and processes used to manufacture and assemble the widgets

  • Materials used - plastic
    - aluminum
    - steel
    - titanium
  • Methods used to form materials - injection molding
    - casting
    - forging
    - machining
  • Methods used to connect sub-components - bonding
    - welding
    - fastened
    - electrical connections

• Moreover, the values measured for each widget will not be the same, but will be uncertain - varying from widget to widget

• This uncertainty will be derived from several sources

  • Impurities in the raw materials used to manufacture the widgets
  • Deviations in the processes used to form the material
  • Imprefections in connecting the sub-components
  • Manufacturing facilities out of calibration

Assessing Uncertainty

• It’s important that we can make certain assurances - “Our widgets will operate without fault or failure for a specified time period”
  - “Our widgets will meet strict dimensional requirements for height, width, weight, etc.” to our customers that the widgets they purchase will meet their requirements.

• To make these assurances we need to collect data on our widgets

• The data will be collected by conducting different types of tests

  • Destructive drop tests to ensure that a typical widget could survive a fall from a designated height
  • Manufacturing verification tests to ensure that the widgets being produced have the correct dimensions

• Each test we perform on a widget generates one or more observational measures (observations)

  • Dimensional measures: “The length of widget-102 is 14.2 inches”
  • Operational measures: “Widget-54 survived the 48-inch drop test”

• Each observation contains information about the properties of the widget being tested

• The information obtained from testing each widget gives us a small glimpse of the properties of the overall population of widgets

• Pooling the information obtained from all of the widgets together provides a more complete view of the population’s properties

Making inferences

• We presume that there exists an underlying mathematical model \(F_{\text{perfect}}\) that perfectly describes the properties of the widgets in the population

• This perfect model would also explain the variability that exists in our observed data due the various sources of uncertainty

  • Uncertainty due to imperfctions in the manufacturing processes
  • Uncertainty due to imperfections in our testing procedures
  • Uncertainty due to imperfections in our measurement capabilities

• We’ll never know the true form of \(F_{\text{perfect}}\) as there are also sources of uncertainty we don’t know about and cannot account for

• Therefore, our goal is to create an mathematical model \(F_{\text{imperfect}}\) that imperfectly represents \(F_{\text{perfect}}\)

  • \(F_{\text{imperfect}}\) will be based on the data we observe and our prior knowledge of the widgets being produced
  • The more information we have the more accurately \(F_{\text{imperfect}}\) will represent \(F_{\text{perfect}}\)
  • More information requires testing more widgets
  • More testing means more time and money spent on testing - and more widgets being tested that cannot be sold to customers

• Questions to consider in collecting data

  • Clearly aren’t going to test every widget that is produced - but, how many should be tested?
  • Out of the population of widgets, how should we choose which widgets to test?

• We want to sample from our population of widgets randomly

  • Capture fluctuations that may occuer in our manufacturing process over time
  • Capture fluctuations in materials used (i.e. changing suppliers)

Random Samples

Defining terms

• The \(n\) observations we collect during a test - denoted as \(\xi = [x_1,\ldots,x_n]\) - are realizations of random variables \(\Xi = [X_1,\ldots,X_n]\)

• When a random sample contains \(n\) realizations \(x_{1},\ldots,x_{n}\) of \(n\) random variables \(\xi = [x_1,\ldots,x_n]\) we say that the sample has size \(n\) (or that the sample size is \(n\)) and an individual realization \(x_{i}\) is referred to as an observation

Incorrect wording: \(n\) independent realizations of the random variable \(X\)

Correct wording: realizations of \(n\) independent random variables \(X_1,\ldots,X_n\)

• In the simplest case the random variables \(X_1,\ldots,X_n\) are independent and have a common distribution function \(F_{_{X}}(x)\) and we say that \(X_1,\ldots,X_n\) are independent and indentically distributed, or IID

• IID Example

  • The lifetime of our widgets is represented by the random variable \(X\), whose distribution function \(F_{_{X}}(x)\) is unknown.
  • Suppose we independently observe the lifetimes of \(10\) widgets and denote these realizations by \(x_{1},x_{2},\ldots,x_{10}\)
  • We’re interested in the expected value of \(X\), which is an unknown characteristic of \(F_{_{X}}(x)\)
  • We infer \(\operatorname{E}[x]\) from the data, and estimate \(\operatorname{E}[x]\) with the sample mean \(\bar{x} = \sum_{i = 1}^{10} x_i/10\)
  • The observed data \(x_{1},x_{2},\ldots,x_{10}\) constitute our sample and \(\operatorname{E}[x]\) is the quantity about which we are making a statistical inference

• While in the simplest case \(X_1,\ldots,X_n\) are independent random variables, more complicated cases are possible

  • \(X_1,\ldots,X_n\) are not independent
  • \(X_1,\ldots,X_n\) are random vectors having a common joint distribution function
  • \(X_1,\ldots,X_n\) do not have a common probability distribution

• In a more general sense we say: A sample \(\xi\) is the realization of a random vector \(\Xi\)

• The distribution function of \(\Xi\), denoted by \(F_{_{\Xi}}(\xi)\), is the unknown distribution function that constitutes the object of inference

• Therefore, “sample” is just a synonym of “realization of a random vector”. The following examples show how this general definition accommodates the special cases mentioned above

Example 1: Independent and Identically Distributed

• We observe \(n\) realizations \(x_{1},\ldots,x_{n}\) of \(n\) independent random variables \(X_1,\ldots,X_n\) having a common distribution function \(F_{_{X}}(x)\)

• The sample is the \(n\)-dimensional vector \(\xi = [x_1,\ldots,x_n]\), which is a realization of the random vector \(\Xi = [X_1,\ldots,X_n]\)

• The joint distribution function of \(\Xi\) is

\[ F_{_{\Xi}}(\xi) = F_{_{X}}(x_1)\times\ldots\times F_{_{X}}(x_n) \]

Example 2: Identically Distributed, but not Independent

• We observe \(n\) realizations \(x_{1},\ldots,x_{n}\) of \(n\) random variables \(X_1,\ldots,X_n\) that are not independent but have a common distribution function \(F_{_{X}}(x)\)

• The sample is again the \(n\)-dimensional vector \(\xi = [x_1,\ldots,x_n]\), which is a realization of the random vector \(\Xi = [X_1,\ldots,X_n]\)

• However, in this case the joint distribution function \(F_{_{\Xi}}(\xi)\) can no longer be written as the product of the distribution functions of \(X_1,\ldots,X_n\)

Statistical Estimation

Overview

• Estimation refers to the use statistical procedures to make inferences about a population, based on information obtained from a random sample

• There are two basic types of statistical procedures

  • Parametric precedures: rely on assumptions about the shape of an underlying probability distribution (i.e. a normal distribution) that describes the population and about the form or parameters (i.e., means and standard deviations) of the assumed distribution
  • Nonparametric models: rely on no or few assumptions about the shape of the underlying distribution or parameters of the population distribution from which the sample was drawn

• In the sections below we formally define statistical models and show ways to compute various types of estimates

• Throughout the remainder of this presentation we’ll assume that parametric models will be used

Statistical Models

• We previously defined a random sample \(\xi\) as a realization of a vector of random variables, \(\Xi\)

  • \(\Xi\) has joint distribution function \(F_{_{\Xi}}(\xi)\)
  • The observed random sample \(\xi\) is used to infer some characteristics of \(F_{_{\Xi}}(\xi)\) that are not already known
  • The properties and the characteristics of \(F_{_{\Xi}}(\xi)\) that are already known before observing the sample are called a model for \(\Xi\).

• Analysts use sample statistics to estimate population parameters

  • The sample mean \(\bar{x}\)The mean computed for a random sample of observed data is used to estimate the unknown mean of the population \(\mu\)

  • The sample variance \(\operatorname{Var}[x]\)The variance computed for a random sample of observed datais used to estimate population proportions.

• An estimate of a population parameter may be expressed in two ways:

  • Point estimates: population parameter is estimated by a single value of a statistic (i.e. the sample mean ${x} = is a point estimate of the population mean \(\mu\))

  • Interval estimate: population parameter is estimated by two numbers, between which the parameter is said to lie (i.e. \(a < \hat{\mu} < b\) is an interval estimate of the population mean \(\mu\)

Point Estimates

Properties of point estimates

• It is desirable that a statistic used as a point estimate for a population parameter be:

  1. Consistent - The larger the sample size, the more accurate the estimate should be
  2. Unbiased - The expectation of the observed values of many samples (“average observation value”) equals the corresponding population parameter
  3. Most efficient - The estimator possessing the smallest variance out of all of the estimators that are consistent and unbiased Note This often depends on the assumed underlying distribution of the population
     
     For example, the mean is more efficient than the median (middle value) for the normal distribution but not for more “skewed” (asymmetrical) distributions.

Methods to compute point estimates

• Several methods are used to calculate the estimator

  • The Bayesian method The Bayesian method differs from traditional methods by introducing a frequency function for the parameter being estimated
    
    A drawback to the Bayesian method is that sufficient information on the distribution of the parameter is usually not available
    
    An advantage is that the estimation can be easily adjusted as additional information becomes available is named for the 18th-century English theologian and mathematician Thomas Bayes
  • The moments method equates values of sample moments (functions describing the parameter) to population moments
  • Maximum likelihood method (most common) uses differential calculus to determine the maximum of the probability function of a number of sample parameters

Interval Estimates

Confidence Intervals

• A type of interval estimate used to express the uncertainty associated with a particular sampling method

• Are preferred to point estimates, because confidence intervals indicate (a) the precision of the estimate and (b) the uncertainty of the estimate

• Consists of three parts.

  • A confidence level: describes the uncertainty of a sampling method
  • A statistic: quantity about which the CI is being generated
  • A margin of error: describes the precision of the method
  • The interval estimate of a confidence interval is defined by the sample statistic + margin of error

• Example: compute an interval estimate of a population parameter

  • We might describe this interval estimate as a \(95\%\) confidence interval
  • This means that if we used the same sampling method to select different samples and compute different interval estimates, the true population parameter would fall within a range defined by the sample statistic + margin of error \(95\%\) of the time.

Confidence Level

• The probability part of a confidence interval is called a confidence level

• Describes the likelihood that a particular sampling method will produce a confidence interval that includes the true population parameter.

• Interpreting a confidence level

  • Suppose we collected all possible samples from a given population, and computed confidence intervals for each sample
  • Some confidence intervals would include the true population parameter; others would not
  • A \(95\%\) confidence level means that \(95\%\) of the intervals contain the true population parameter

Margin of Error

• In a confidence interval, the range of values above and below the sample statistic is called the margin of error.

• Interpreting the margin of error

  • Suppose a newspaper conducts an election survey and reports that a candidate will receive 30% of the vote
  • The newspaper states that the survey had a 5% margin of error and a confidence level of \(95\%\)
  • These statements implies a confidence interval for which we are \(95\%\) confident that the candidate will receive between \(25\%\) and \(35\%\) of the vote

Hypothesis Testing

Overview

• Suppose a population of observations exists to which no treatment has been applied

  • We call this the “comparison group” of the “control group”
  • We know the parameters associated with this population (i.e. the mean and/or standard deviation).

• Now suppose that another population exists that’s similar to the control group, except that some treatment has been applied

  • We call this the treatment group or experimental group
  • You do not know the parameters associated with this population

• Samples are drawn from this latter population and the statistics derived from the sample serve as the estimates of the unknown population parameters

• A hypothesis statement is created to describe the statistical relationship between the two data sets - this statement contains two elements:

  • A Null hypothesis \(H_O\) proposes that a statistically significant relationship does exist between the two data sets
  • An alternative hypothesis \(H_A\) proposes that a statistically significant relationship does not exist between the two data sets

• The comparison is deemed statistically significant if the relationship between the data sets would be an unlikely realization of the null hypothesis according to a threshold probability – the significance level

• Hypotheses are helpful in determining what outcomes of a study would lead to a rejection of the null hypothesis for a pre-specified level of significance

• Distinguishing between the null and alternative hypotheses is aided by considering two types of errors

  • Type 1 error: Occurs when the null hypothesis is wrongly rejected
  • Type 2 error: Occurs when the null hypothesis is wrongly not rejected

• Hypothesis tests based on statistical significance are akin to expressing confidence intervals

  • Every hypothesis test based on significance can be obtained via a confidence interval
  • Every confidence interval can be obtained via a hypothesis test based on significance

Hypothesis Testing Steps

• Step 1) Develop the research statement (hypothesis). That is, state what you hope to discover. Interrogative form is common.

  • Is this hypothesis testable? As attractive as it may be, if a hypothesis cannot be tested, it has little utility to empirical investigators.

  • Hypotheses are not testable if the concepts (constructs) to which they refer are not adequately (operationally) defined. To facilitate clarity in communication, operational definitions (procedures that define variables) provide the concrete means of evaluating whether a hypothesis constains concepts that can be observed.

  • Hypotheses are also untestable if they are circular. That is, the event or outcome becomes an explanation for the event. For example: “Your eight-year-old son is distractable in school because he has attention deficit hyperactivity disorder.”

• Step 2) Set up the null hypothesis (\(H_0\))

  • Hypothesis that assumes no difference between treatment and comparison groups – that the treatment has no effect
  • Note that this assumption is the exact opposite of what you wish to discover

• Step 3) Construct the sampling distribution based on the assumption that \(H_0\) is true

  • Choose a particular test statistic (e.g., \(z\), \(t\), \(F\)) based on particular assumptions about the population
  • Draw a sample, and collect data

• Step 4) Compare the sample statistic to the the distribution

  • What is likelihood of obtaining the observed result if the null hypothesis is true?

• Step 5) Reject or fail to reject the \(H_0\) depending on the probability

  • If the likelihood of the observed difference between means is small we reject the null hypothesis and conclude that the treatment did have an effect
  • When the difference is large, and hence unlikely due to chance variation, we fail to accept the null hypothesis. Then the difference is said to be “statistically significant.”

Specialized Parametric Hypothesis tests

• Several parametric tests exist (e.g., \(z\), \(t\), \(F\), \(\chi^2\))

  • These tests require the data meet certain assumptions before the test can be performed
  • If the data do not meet these assumptions the results are, at best, meaningless, and at worst, misleading

Summary

In this presentation

• Discussed that inference is a decision making process

  • Uses statistics and probability theory
  • Used to describe a population based on a random sample of observations drawn from the population

• We introduced the types of inference

  • Estimation
  • Hypothesis testing

In the next presentation

• Walkthrough examples of carrying out various type of inferences

  • Maximum likelihood estimation of population parameters
  • Create hypothesis statements
  • Use Python to conduct \(t\), \(F\), \(\chi^2\) hypothesis tests